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I.  Introduction 


This  document  describes  the  equipment  purchased  under  the  Defense  University 
Research  Instrumentation  Program  awarded  in  1997  to  Nancy  J.  Cooke  of  the  Psychology 
Department  of  New  Mexico  State  University.  The  equipment  is  housed  in  a  facility 
called  the  CERTT  Lab  (Cognitive  Engineering  Research  on  Team  Tasks)  which  is 
located  in  Room  265  of  Science  Hall  at  New  Mexico  State  University.  The  lab  consists 
of  four  NT-based  participant  workstations  and  an  NT  experimenter  control  workstation. 
The  hardware  is  configured  to  support  a  variety  of  synthetic  task  environments, 
experimental  requirements,  and  data  collection  and  analyses  modules.  The  initial 
software  design  focuses  on  a  synthetic  task  that  preserves  many  of  the  cognitive  features 
of  UAV  (unmaimed  air  vehicle)  operation,  in  addition  to  modules  for  experimental 
control  and  data  collection  and  analyses.  Software  development  is  ongoing  with  iterative 
testing  and  design  cycles  guiding  the  implementation  of  initial  software  design  plans. 

The  purpose  of  the  CERTT  lab  is  to  create  a  flexible  research  enviromnent  for  the  study 
of  team  cognition.  In  particular,  measures  of  knowledge  at  the  team  level  will  be 
developed  and  evaluated  in  this  setting.  Functionality  requirements  for  the  lab  were 
determined  through  a  needs  analysis  of  projected  laboratory  users  including  researchers 
in  NMSU’s  Psychology  Department,  Armstrong  Lab,  and  the  Naval  Air  Warfare  Center- 
Training  Systems  Division.  This  analysis  indicated  that  while  high  fidelity  simulations  of 
the  UAV  task  (and  other  team  tasks)  existed  (e.g.,  at  Williams  AFB)  what  was  still 
needed  was  a  researcher-friendly  facility  that  embodied  a  low-fidelity  synthetic  task  that 
would  permit  various  forms  of  experimental  control,  measurement,  and  a  broad  spectrum 
of  participants 

Therefore  the  workstations  and  software  in  the  CERTT  lab  permit  the  simulation  of  a 
variety  of  complex  team  tasks.  However,  a  major  emphasis  is  placed  on  the  functions 
required  by  experimenters  to  conduct  research  in  such  a  setting,  such  as  the  ability  to  alter 
task  parameters  and  record  data  from  a  variety  of  sources.  The  CERTT  Lab  is  also 
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described  in  a  newspaper  clip  from  the  local  Las  Cruces  Sun  News  (August,  1997) 
reproduced  in  Figure  1. 
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Figure  1.  Press  release  on  the  CERTT  Lah,  August,  1997. 


The  remainder  of  this  document  describes  the  CERTT  laboratory  from  three 
perspectives; 

□  laboratory  layout, 

□  workstation  design,  and 

□  software  design. 
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II.  Laboratory  Layout 


The  following  diagrams  represent  the  layout  of  the  equipment  and  signage  within  the 
CERTT  Lab.  The  lab  space  includes  one  large  room  and  three  smaller  rooms  to  the  side, 
totaling  approximately  600  square  feet.  The  four  participant  workstations  are  currently 
arranged  as  indicated  in  Option  A.  These  are  single-participant  workstations  and  the 
height  of  each  workstation  restricts  participants’  views  of  each  other.  Portable  partitions 
can  be  easily  installed  if  necessary  to  provide  additional  participant  isolation.  This 
feature,  combined  with  the  use  of  headsets  for  communication  allows  for  task  scenarios 
in  which  team  members  interact  from  isolated  locations.  However,  arranging  the 
workstations  as  shown  in  Option  B  allows  for  scenarios  in  which  pairs  of  team  members 
are  co-located. 


Figure  2.  CERTT  Floor  Plan  -  Console  Layout  Option  A  (drawn  to  scale). 
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Figure  3.  CERTT  Floor  Plan  -  Console  Layout  Option  B  {drawn  to  scale). 


SOUTH  VIEW 

Figure  4.  CERTT  Laboratory  South  Wall  View 
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EAST  VIEW 


Figure  5.  CERTT  Laboratory  East  Wall  View 


WEST  VIEW 

Figure  6.  CERTT  Laboratory  West  Wall  View 
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III.  Workstation  Design 


The  CERTT  Lab  consists  of  four  participant  workstation  and  one  experimenter 
workstation.  Each  of  the  participant  workstations  consists  of  two  computers  and 
monitors,  a  video  monitor,  a  joystick,  a  keyboard  and  mouse,  a  headset  and  intercom, 
along  with  a  variety  of  other  lights,  power  switches,  and  indicators.  These  devices  are 
configured  in  each  of  the  three-sided,  90°,  workstations  as  indicated  in  the  following 
diagrams.  The  consoles  are  manufactured  by  Winsted  Industries  and  are  similar  to  those 
used  in  the  video  production  industry  and  television  studios.  The  primary  computer  is  a 
NT-based  system  having  a  17-inch  monitor.  The  secondary  computer  is  a  rack 
mountable  industrial  computer  produced  by  Cyber  Research.  This  computer  is  used 
primarily  for  monitoring,  tracking,  and  various  secondary  tasks,  whereas  the  primary 
computer  displays  most  of  the  main  information  for  the  synthetic  task.  The  RGB  video 
monitor,  a  rack-mount  Sony  unit,  allows  the  presentation  of  video  training  or  task 
materials.  On  the  experimenter  console  the  video  monitor  permits  the  experimenter  to 
view  each  of  the  four  participants  via  a  Sony  video  camera  mounted  to  the  ceiling  behind 
each  of  the  four  participants.  The  lab  design  allows  the  monitoring  cameras  to  be  easily 
moved.  The  joystick  is  a  military  aircraft  styled  unit  and  allows  for  realistic  flight-related 
interactions  with  the  system.  Care  has  been  taken  in  the  design  of  the  workstations,  to 
simplify  the  functions  available  to  the  user  and  to  disable  any  functions  that  are  not 
necessary  or  desirable  for  the  user  to  perform  (e.g.,  turning  off  the  main  power.) 

Careful  planning  went  into  the  communication  module  of  the  workstations.  The  headsets 
and  intercoms  provide  direct  experimenter  control  of  various  communication  pathways 
among  team  members.  The  experimenter  specifies  those  paths  that  are  viable.  In 
addition,  audio  data  will  be  recorded  in  sync  with  other  data  records  (i.e.,  time-stamped) 
and  will  identify  not  only  time  of  communication,  but  speaker’s  and  listener’s  identities 
as  well.  The  communication  module  also  creates  auditory  isolation  of  team  members 
when  necessary. 
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One  ASL  head  tracker  will  be  included  in  the  CERTT  Lab.  This  allows  for  recording  of 
the  position  of  the  team  member’s  head  at  a  finer  level  than  can  be  obtained  fi’om  video 
records.  It  was  determined  that  for  workstations  so  big,  that  an  eye-tracker-level  of 
resolution  was  not  justified  nor  desirable  because  of  the  wide  angle  of  view  (90°)  of  the 
participant  workstation.  The  four  participant  workstations  and  the  experimenter  station 
are  connected  locally  using  a  100BaseT  local  area  network.  Additionally,  the  system  can 
be  readily  connected  to  the  internet  via  the  NMSU  network 


Top  is  12  RUs 


Workbench  Sipport  Area 

Bottom  is  14  RUs 


Figure  7.  CERTT  Workstation  Console  Layout 


Rack  A  Rack  B  Rack  C 


Rack  F 

Locking  Plexiglass  Door 
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rV.  Software  Design 


Software  design  for  this  project  continues  by  Sandia  Research  Corporation  at  no 
additional  cost  to  this  project.  The  careful  planning  that  went  into  the  construction  of  the 
workstations  described  above  has  laid  the  ground-work  to  support  a  wide  variety  of 
synthetic  task  scenarios  and  software  modules  to  support  experimentation.  These 
modules  and  their  functions  are  described  in  Section  A  below.  Software  modifications 
will  continue  with  further  test  and  redesign  iterations.  The  first  pass  at  the  synthetic  task 
itself  is  outlined  in  Section  B.  This  design  too  is  continually  fme-tvmed  with  feedback 
from  test-design  iterations. 

A.  Software  Modules  &  Functionality 

1.  Event  logger 

-  records  user  input  to  system  (keystrokes,  mouse  clicks,  joystick  information) 

-  records  any  dynamic  (i.e.,  unplanned)  events/messages  presented  to  user 

-  timestamps  system  events  and  user  events  (in  sync  with  video  and  audio  data) 

-  experimenter  control  of  level  of  abstraction  of  recorded  events 
flexible  summary  format 

-  interfaces  with  software  tools  for  event  analyses 

2.  Training  module 

-  capability  of  presenting  different  training  materials  to  each  individual 

-  incorporates  testing  procedures  (see  factual  test  module) 

-  allows  some  material  to  be  referenced  throughout  task 


Training  Module 


Figure  8.  Software  System  Training  Component 
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Knowledge  elicitation  modules 

-  Relatedness  Ratings:  This  module  takes  a  set  of  concepts  and  randomly  presents 
pairs  of  these  concepts,  counterbalancing  the  order  of  the  first  item  in  the  pair 
across  subjects.  Pairs  are  presented  on  the  computer  screen  with  a  rating  scale. 
Relatedness  ratings  are  collected  and  summarized  in  a  format  that  interfaces  with 
analytic  tools  like  Pathfinder 

-  Diagramming  Module:  This  module  allows  the  participant  to  create  node-link 
diagrams  in  which  nodes  and  links  can  be  assigned  verbal  labels,  links  can  be 
drawn  in  one  or  both  directions,  and  links  and  nodes  can  be  annotated  according 
to  strength.  Data  are  summarized  in  a  form  compatible  with  analytic  tools  like 
Pathfinder. 

-  Factual  Tests:  This  module  randomly  presents  test  questions  or  surveys,  collects 
the  responses,  and  in  the  case  of  close-ended  questions,  scores  responses  and  is 
capable  of  providing  immediate  feedback  when  required.  Data  are  recorded  and 
summarized  in  format  specified  by  experimenter. 


Figure  9.  Software  Knowledge  Elicitation  Component 


3.  Task  Modules 

-  Embedded  Measures:  Task  can  be  interrupted  to  present  queries;  but  in  many 
cases  these  measures  are  collected  as  part  of  the  task  (e.g.,  task  outcome, 
performance  (time,  accuracy),  situation  awareness). 

-  Experimenter  Annotations:  Experimenter  has  capability  of  annotating  aspects  of 
team  interactions  occurring  throughout  the  task.  These  data  are  summarized  and 
time-stamped  so  that  they  are  in  sync  with  other  records  of  task  activities. 

-  Experimenter  Control:  The  task  has  parameters  that  can  be  controlled  by 
experimenter  (e.g.,  degree  of  complexity,  coordination  demands,  degree  of  team 
interdependence,  degree  of  ambiguity,  time  pressure,  presence  of  conflict, 
severity  of  consequences,  framing  of  problem,  presence  of  dynamic  events).  The 
experimenter  also  has  the  ability  to  alter  these  parameters  at  the  level  of  the 
individual  console. 

-  Task-Specific:  Modules  such  as  briefing,  planning,  navigation  aspects  of  the  task 
are  designed  in  modules  so  that  the  task  can  maximally  re-configurable  (e.g.,  for 
other  synthetic  tasks  such  as  air  traffic  control,  flight  crew  missions). 
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Figure  10.  Software  Synthetic  Task  and  Event  Logger 


B.  The  UAV  Synthetic  Task 

The  needs  analysis  indicated  that  the  synthetic  task  should  be  cognitively  complex.  In 
other  words,  it  should  require  cognitive  team  activities  such  as  team  decision  making, 
team  situation  awareness,  knowledge  sharing,  planning,  and  communication.  It  was  also 
important  that  it  have  features  associated  with  team  interaction  such  as  distributed 
expertise,  team  member  interdependence,  dynamic  interplay  between  team  members,  and 
the  sharing  of  scarce  resources.  Finally,  it  was  also  critical  that  the  task  be 
understandable  and  be  engaging  for  college  undergraduates. 

With  these  objectives  in  mind  a  UAV  scenario  to  serve  as  the  backbone  of  the  initial 
synthetic  task  was  scripted.  An  outline  of  the  scenario  is  presented  below. 

Initial  UAV  Scenario 
Background  for  all  team  members: 

There  have  been  intelligence  reports  that  the  cite  of  a  small  college  in  Western 
Australia  has  now  become  a  center  for  nuclear  weapons  which  are  to  be  sold  to 
Iraq.  Your  mission  is  to  bring  back  aerial  photos  of  this  cite. 
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During  the  scenario  events  are  subject  to  change.  These  events  include  weather 
conditions,  aircraft  malfunctions,  payload  malfunctions,  and  enemy  activities. 

Roles  of  three  team  members: 

1)  A  VO  (Air  Vehicle  Operator): 

Responsibility:  Is  responsible  for  operating  an  unmanned  air  vehicle  between 
Phoenix,  AZ  and  Western  Australia. 

Individual  Goals:  Take-off,  landing,  and  safe  operations  of  UAV.  A  VO  is  also 
responsible  for  detecting  any  equipment  malfunctions,  diagnosing  them,  and  taking 
appropriate  action. 

Training:  A  VO  receives  individual  training  in  using  a  flight  simulator  program. 
Information  Available:  Lists  of  common  malfunctions,  diagnoses,  and  recommended 
actions,  computer  also  displays  indications  of  malfunctions 
Actions  Available:  Basic  flight  operations,  maintenance  actions  (checks,  repairs) 


2)  Mission  Commander. 

Responsibility:  Responsible  for  overseeing  mission,  route  planning,  making  any 
necessary  changes  in  mission. 

Individual  Goals:  Navigation  and  route  planning.  Accomplish  mission  while 
minimizing  flight  time  and  ensuring  safety  of  the  equipment  and  other  team  members. 
Facilitate  team  coordination. 

Training:  Will  receive  training  in  team  coordination  (planning,  communication).  In 
addition  will  receive  training  in  navigation  (map  reading,  GPS  use). 

Information  Available:  Details  of  mission  and  background,  including  a  map  of 
various  routes  and  details  associated  with  each  (i.e.,  terrain,  enemies,  mileage,  etc.). 
Will  also  get  periodic  weather  reports  via  computer  and  GPS  information. 

Actions  Available:  Route  information  can  be  selected  which  is  then  sent  to  A  VO 


3)  Payload  Operator: 

Responsibility:  Responsible  for  taking  pictures  remotely  of  designated  cite,  also 
responsible  for  identifying  potential  threats  to  aircraft  from  enemies. 

Individual  Goals:  Identify  target,  direct  camera,  take  picture,  zoom  as  needed,  predict 
potential  threats  and  communicate  them  to  the  Mission  Commander. 

Training:  receives  training  in  the  use  of  the  camera  and  identifying  and  predicting 
enemy  operations. 

Information  Available:  Cues  that  help  identify  designated  cite  (e.g.,  photo  of 
surrounding  area),  intelligence  information  about  potential  covert  enemy  operations 
(both  on  paper  and  during  task  via  computer) 

Actions  Available:  direct  camera,  zoom,  take  pictures,  query  an  intelligence  database 


12 


V.  Conclusion 


The  CERTT  Laboratory  offers  a  flexible  configuration  of  hardware  and  software  to 
support  the  development  of  a  variety  of  synthetic  tasks.  The  lab  is  particularly  suited 
to  the  implementation  of  synthetic  team  tasks  requiring  information  sharing, 
communication  and  coordination  among  two  to  four  team  members.  Most 
importantly,  the  CERTT  Lab  provides  a  research  environment  for  the  study  of  team 
cognition.  Hardware  and  software  permit  flexible  data  recording  and  summarization 
from  a  variety  of  sources  including  video,  audio,  computer,  and  head  tracking 
equipment.  In  addition,  the  experimenter  has  a  variety  of  software  options  for 
experimental  control.  This  experimental  control  and  data  recording  flexibility  in 
conjunction  with  the  synthetic  team  task  environment  make  the  CERTT  Laboratory  a 
unique  and  valuable  research  tool  for  the  study  of  team  cognition. 
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